The control system of autonomous vehicles can sometimes be configured using a simulated human driver environment. The simulated human driver environment attempts to model the typical driving behavior of human drivers in various driving environments. However, the simulated human driver environment may be built based on the information gathered from the sensors and cameras on the autonomous vehicle or related probe vehicles. Because this information, including images from the vehicle cameras, can be subject to image occlusion and unreliable image reconstruction accuracy, the utility and effectiveness of the simulated human driver environment is degraded. Additionally, the image occlusion problem is further complicated by shadows cast by both the vehicles themselves and overhead objects such as trees, buildings, construction equipment, and the like. Classic color-based methods for shadow detection or removal do not work in this case; because of the diversity of vehicle types and colors. Because of these problems with the image data gathered for configuring the simulated human driver environment, the effectiveness of the control systems of autonomous vehicles based on the degraded simulated human driver environment can be compromised.